A Novel Analytical Approach for Lip Synchronization

We present a novel approach for Lip synchronization by analyzing the relationship between a person’s speech signal and data extracted from his/her lip movements. To model the speech we use a nonlinear-time-varying sum of AM-FM signals each of which models a single formant frequency. The model is then realized using Taylor series expansions such that a closed form formula is achieved which shows the relationship between the speech amplitudes and instantaneous frequencies w.r.t lips varying width and height. Based on the obtained formula, lips movements data are employed to generate a semi-speech signal which is then correlated with the original speech over a span of delays. From the resultant correlation, the delay between the two signals is estimated, hence Lip Sync is achieved. The approach is applied to practical speech examples and the obtained results support the correctness and consistency of our proposed approach. The developed method can estimate delays around 0.1 second at low SNRs and 0.04 second at high SNRs.